Methods and systems of organizing vendors of production print services by ratings

ABSTRACT

Methods and systems of evaluating customer satisfaction with a plurality of print-related items are disclosed. A plurality of evaluator ratings, each including an ordinal scale value, may be received for each print-related item in a set. A rating distribution may be determined for each print-related item based on the received evaluator ratings. A similarity value between the rating distribution for the print-related item and each other print-related item may also be determined. The set of print-related items may be divided into one or more clusters based on the corresponding similarity values. A comparison of the one or more clusters may be displayed. The comparison may include a comparison of the similarity values and/or a rating variability between rating distributions associated with the print-related items in one or more clusters and/or a comparison of average rating distributions associated with one or more clusters.

BACKGROUND

Customers are typically asked to assess the quality or effectiveness ofa product or service through customer surveys, Web-based feedback formsor the like. Typically, customers are asked to rate a product or serviceusing a scale, such as from zero to five. The average rating is usuallydetermined by summing the ratings and dividing the sum by the totalnumber of evaluators who rated the product or service. Determining theaverage in this manner can mislead customers as to the quality of theproduct or service, however, especially when the rating distribution isbimodal, or has a large number of low ratings and a large number of highratings. For example, a product that receives two ‘1’ ratings and two‘5’ ratings (on a scale from one to five) has the same average as aproduct that receives four ‘3’ ratings.

In particular, if a product or service is offered by multiple vendors,consumers may rate the product or service very differently based on thevendor who provides the product or service. As such, it would be helpfulto a potential consumer of a good or service if useful information wereavailable regarding the effect of a vendor on consumer ratings. Forexample, a seller of printing devices may contract with several vendorsto provide maintenance services on those devices. While the services maybe identical, consumer satisfaction with the service may vary widely byvendor. If a method or system were available to assess consumersatisfaction of a service by vendor and compare the data of one vendorto others, the print device supplier could address issues withunderperforming maintenance service vendors and thus improve customersatisfaction.

A broker may offer production print services including printing,binding, envelope insertions, three-hole drilling, etc. on behalf ofclients. Production print services are contracted through a number ofvendors, Clients may rate the performance of vendors according to anumber of criteria including product quality, ability to satisfy servicelevel agreements, and value. A broker would like to offer clients thebest possible service and value by assessing vendor ratings and organizevendors into tiers or clusters of ones with similar ratingsdistributions.

Methods and systems for clustering and displaying product and serviceratings by rating distribution would be desirable.

SUMMARY

Before the present methods are described, it is to be understood thatthis invention is not limited to the particular systems, methodologiesor protocols described, as these may vary. It is also to be understoodthat the terminology used herein is for the purpose of describingparticular embodiments only, and is not intended to limit the scope ofthe present disclosure which will be limited only by the appendedclaims.

It must be noted that as used herein and in the appended claims, thesingular forms “a,” “an,” and “the” include plural reference unless thecontext clearly dictates otherwise. Thus, for example, reference to arating is a reference to one or more ratings and equivalents thereofknown to those skilled in the art, and so forth. Unless definedotherwise, all technical and scientific terms used herein have the samemeanings as commonly understood by one of ordinary skill in the art. Asused herein, the term “comprising” means “including, but not limitedto.”

In an embodiment, a method of evaluating customer satisfaction with aplurality of print-related items may include receiving a plurality ofevaluator ratings for each print-related item in a set of print-relateditems, where each rating comprises an ordinal scale value. For eachprint-related item, a rating distribution may be determined by acomputing device for the print-related item based on the receivedevaluator ratings. A similarity value between the rating distributionfor the print-related item and for each other print-related item in theset of print-related items may be determined by a computing device. Theset of print-related items may be divided into one or more clustersbased on the corresponding similarity values. Each cluster may includeone or more print-related items. A comparison of the one or moreclusters may be displayed to a user. The comparison may include one ormore of a comparison of the similarity values between one or more ratingdistributions associated with the one or more print-related items in oneor more clusters, a comparison of rating variability between one or morerating distributions associated with the one or more print related itemsin one or more clusters, and a comparison of one or more average ratingdistributions associated with one or more clusters.

In an embodiment, a system of evaluating a plurality of print-relateditems may include a processor, a communication port in communicationwith the processor and a processor-readable storage medium incommunication with the processor containing one or more programminginstructions for performing a method of evaluating customer satisfactionwith a plurality of print-related items. The method may includereceiving a plurality of evaluator ratings for each print-related itemin a set of print-related items, where each rating comprises an ordinalscale value. For each print-related item, a rating distribution may bedetermined by a computing devices for the print-related item based onthe received evaluator ratings. A similarity value between the ratingdistribution for the print-related item and for each other print-relateditem in the set of print-related items may be determined by a computingdevice. The set of print-related items may be divided into one or moreclusters based on the corresponding similarity values. Each cluster mayinclude one or more print-related items. A comparison of the one or moreclusters may be displayed to a user. The comparison may include one ormore of a comparison of the similarity values between one or more ratingdistributions associated with the one or more print-related items in oneor more clusters, a comparison of rating variability between one or morerating distributions associated with the one or more print related itemsin one or more clusters, and a comparison of one or more average ratingdistributions associated with one or more clusters.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an exemplary chart of products and corresponding ratingsaccording to an embodiment,

FIG. 2 depicts exemplary rating distributions for products based onexemplary ratings illustrated by FIG. 1.

FIG. 3 illustrates exemplary values used in calculating the Mallowsdistance between two probability distributions according to anembodiment.

FIG. 4 depicts an exemplary dendrogram of clusters formed usinghierarchical agglomerative clustering according to an embodiment.

FIG. 5 depicts an exemplary graph of probability distributions forevaluated products according to an embodiment.

FIG. 6 depicts an exemplary flow chart of evaluating a plurality ofitems according to an embodiment.

FIG. 7 depicts a block diagram of exemplary internal hardware that maybe used to contain or implement the program instructions according to anembodiment.

DETAILED DESCRIPTION

Consumers of a certain item may be asked to evaluate the item by ratingit on an ordinal scale. Items may include print-related items such asprinting products, printing services and the like. Printing products mayinclude printing devices such as printers, cutters, collators, bindersand the like. Printing services may include document productionservices, document production maintenance services and the like. Anordinal scale may include numbers, letters, symbols or the like used toassign ranks to items. For example, a consumer of document productionservices may be asked to rate a service that is available from one ormore document production vendors using an ordinal scale that includesvalues from one to five, with one representing the lowest rating andfive representing the highest rating. Similarly, a seller of printingdevices may contract with several vendors to provide maintenanceservices on those devices, for example. The seller, as well as otherconsumers of the maintenance services, may rate their satisfaction withthe provided service.

In an embodiment, consumers may be asked to rate a product or service ona scale from zero to five, with one representing the lowest rating, fiverepresenting the highest rating and zero representing a non-response, aninapplicable response or the like. Other values or rating scales may beused. FIG. 1 illustrates an exemplary chart of ratings provided by tenevaluators, E1-E10 100 corresponding to three products, A 105, B 110 andC 115.

In an embodiment, a rating distribution may be determined for eachproduct based on the evaluator ratings. For each product, the number ofconsumers who selected each rating may be determined. A probabilityvalue may then be determined for each rating by dividing the number ofevaluators who selected each rating by the total number of evaluatorswho rated the product.

In an embodiment, the lowest rating, in this case, ‘0’ ratings, may beremoved from a probability value calculation. For example, threeevaluators may rate a product. The first evaluator may assign a ‘2’rating to the product, the second evaluator may assign a ‘3’ rating tothe product, and the third evaluator may assign a ‘0’ rating to theproduct. An evaluator may use a ‘0’ rating if the evaluator has noexperience with the product being evaluated, if the evaluator wouldprefer not to rate the product or the like. A probability value may notbe calculated for a ‘0’ rating, and the total number of evaluators usedto determine probability values for other ratings may be reduced by thenumber of evaluators who assigned a ‘0’ rating. For example, in theabove example, the probability value associated with the ‘2’ rating maybe 0.5 because 0.5 equals the number of evaluators who selected the ‘2’rating for the product (i.e., 1) divided by the total number ofevaluators who rated the product less the number of evaluators whoassigned a ‘0’ rating to the product (i.e., 2).

FIG. 2 illustrates exemplary rating distributions for product A 200,product B 205 and product C 210 based on the ratings illustrated byFIG. 1. For example, referring back to FIG. 1, one evaluator (i.e., E1)assigned a ‘1’ rating to product A 105, one evaluator (i.e., E9)assigned a ‘3’ rating to product A, six evaluators (i.e., E2, E4, E5,E6, E8 and E10) assigned a ‘4’ rating to product A and two evaluators(i.e., E3 and E7) assigned a ‘5’ rating to product A. Evaluators E1-E10100 rated product A, so the total number of evaluators is ten. As such,the probability associated with the ‘1’ rating for product A may be 0.1215 because 0.1 is equal to the number of consumers who selected the ‘1’rating for product A 200 (i.e., 1) divided by the total number ofconsumers who evaluated product A (i.e., 10). FIG. 2 illustrates therating distributions for product B 205, product C 210 and the remainderof product A 200.

In an embodiment, the similarity between two ratings distributions maybe determined using measures such as Jensen-Shannon divergence,Euclidean distance, Mallows distance or the like.

Jensen-Shannon divergence measures the similarity between twoprobability distributions, such as the ratings distributions illustratedin FIG. 2. If p represents a first probability distribution, and qrepresents a second probability distribution, then the Jensen Shannondivergence between p and q is represented by:

JS(p,q)=H(α*p+(1−α)*q)−α*H(p)−(1−α)*H(q),0≦α≦1

-   -   where H(p) represents the entropy of p and is represented by:

H(p)=−Σ_(i=1) ^(n) p ₁ log(p ₁)

For example, p may represent the rating distribution for product Ahaving values p=(0.1, 0.3, 0.2, 0.4), and q may represent the ratingdistribution for product B having values q=(0.4, 0.0, 0.5, 0.1). Thesimilarity between the distributions of product A and product B may bedetermined using the above Jensen-Shannon divergence calculation. Forexample, if

${\alpha = {1/2}},{{{JS}( {p,q} )} = {{{H( {{\frac{1}{2}*p} + {\frac{1}{2}*q}} )} - {\frac{1}{2}*{H(p)}} - {\frac{1}{2}*{H(q)}\mspace{14mu} {and}\mspace{14mu} {H(p)}}} = {- {\sum\limits_{i = 1}^{n}{p_{i}\mspace{11mu} \log \mspace{11mu} {( p_{i} ).}}}}}}$

As such, the following values may be determined:

$\begin{matrix}{{{\frac{1}{2}*p} + {\frac{1}{2}*q}} = {( {0.05,0.15,0.1,0.2} ) + ( {0.2,0.0,0.25,0.05} )}} \\{= ( {0.25,0.15,0.35,0.25} )}\end{matrix}$ $\begin{matrix}{{H( {{\frac{1}{2}*p} + {\frac{1}{2}*q}} )} = {{{- 0.25}*{\log (0.25)}} - {0.15*\log (0.15)} -}} \\{{{0.35*{\log (0.35)}} - {0.25*{\log (0.25)}}}} \\{= 1.345153}\end{matrix}$ $\begin{matrix}{{H(p)} = {{{- 0.1}*{\log (0.1)}} - {0.3*{\log (0.3)}} - {0.2*}}} \\{{{\log (0.2)} - {0.4*{\log (0.4)}}}} \\{= 1.279854}\end{matrix}$ $\begin{matrix}{{H(q)} = {{{- 0.4}*{\log (0.4)}} - {0.0*{\log (0.0)}} - {0.5*}}} \\{{{\log (0.5)} - {0.1*{\log (0.2)}}}} \\{= 0.9433484}\end{matrix}$ $\begin{matrix}{{{H( {{\frac{1}{2}*p} + {\frac{1}{2}*q}} )} - {\frac{1}{2}*{H(q)}}} = {1.345153 - {1\text{/}2(1.279854)} -}} \\{{1\text{/}2(0.9433484)}} \\{= 0.065299}\end{matrix}$

As such, the similarity between the distributions of product A andproduct B may be represented by the value 0.065299.

In an embodiment, α may represent a weight used to determine similaritybetween two distributions. The weight may be determined based on thetype of evaluator who assigns ratings to a product or service. Forexample, if the ratings of a first rating distribution were provided bythe ordinary consumers, the α associated with the first ratingdistribution may have a value of ½. However, if the ratings of secondrating distribution were provided by experts, frequent purchasers, highspenders or the like, then the α associated with the second ratingdistribution may have a higher value such as ⅔.

In an embodiment, Euclidean distance may be used to measure the distancebetween two probability distributions. If the probability distributionfor product A is p=(p₁, p₂, . . . , p_(n)) and the probabilitydistribution for product B is q=(q₁, q₂, . . . , q_(n)), then theEuclidean distance between p and q is defined as:

$\begin{matrix}{{{ED}( {p,q} )} = \sqrt{( {p_{1} - q_{1}} )^{2} + ( {p_{2} - q_{2}} )^{2} + \ldots + ( {p_{n} - q_{n}} )^{2}}} \\{= \sqrt{( {\sum\limits_{i = 1}^{n}( {p_{i} - q_{i}} )^{2}} )}}\end{matrix}$

If p=(0.1, 0.3, 0.2, 0.4) and q=(0.4, 0.0, 0.5, 0.1), then the Euclideandistance between the distributions of product A and product B may berepresented by:

$\begin{matrix}{{{ED}( {p,q} )} = \sqrt{( {0.1 - 0.4} )^{2} + ( {0.3 - 0.0} )^{2} + ( {0.2 - 0.5} )^{2} + ( {0.4 - 0.1} )^{2}}} \\{= 0.6}\end{matrix}$

In an embodiment, a Mallows distance may measure the similarity betweentwo probability distributions. The Mallows distance may be used todetermine the similarity for probability distributions of two productsor services rated according to different scales.

For example, product A may be rated on a scale from 1 to 5, and productB may be rated on a scale from 1 to 6. Exemplary rating counts forproduct A may be (6, 7, 8, 0, 1) if six people assigned a ‘1’ rating toproduct A, seven people assigned a ‘2’ rating to product A and so on.For a product B rated on a scale from 1 to 6, exemplary rating countsfor product B may be (4, 5, 2, 3, 3, 10) where four people assigned a‘1’ rating to product B, five people assigned a ‘2’ rating to product Band so on.

The corresponding probability distributions may be determined bydividing the number of evaluators who assigned a specific rating to aproduct by the total number of evaluators who rated that product.

If a ratings count for product A is represented by X=(x₁, . . . , x_(m))and a ratings count for product B is represented by Y=(y₁, . . . ,y_(n)), where m and n are possibly not equal, then:

${p_{i} = \frac{x_{i}}{N_{x}}},{i = 1},\ldots \mspace{11mu},m$${q_{i} = \frac{y_{i}}{N_{y}}},{i = 1},\ldots \mspace{11mu},n$

-   -   where α≧1,

N _(x) =x ₁ +x ₂ + . . . +x _(m),

N _(y) =y ₁ +y ₂ + . . . +y _(n)

For example, the probability associated with rating ‘1’ for product Amay be 0.27 because 0.27 equals the number of evaluators who assigned a‘1’ rating to product A (i.e., 6) divided by the total number ofevaluators of product A (i.e., 22). As such, the probabilitydistribution for product A may be represented by p=(0.27, 0.32, 0.36,0.0, 0.05). Similarly, the probability distribution for product B may berepresented by q=(0.15, 0.19, 0,07, 0.11, 0.37).

The Mallows distance between the distributions for product A and productB may be represented by:

${d_{Mallows}( {X,{Y;\alpha}} )} = {\min\limits_{f_{ij}}{\sum\limits_{i = 1}^{m}{\sum\limits_{j = 1}^{n}{f_{ij}{{x_{i} - y_{j}}}^{\alpha}\mspace{14mu} {such}\mspace{14mu} {that}}}}}$? ?indicates text missing or illegible when filed

For this particular example, the f-values for the solution may berepresented by Table 1:

TABLE 1 j i 1 2 3 4 5 1 0.00 0.00 0.15 0.00 0.00 2 0.00 0.10 0.09 0.000.00 3 0.00 0.06 0.01 0.00 0.00 4 0.05 0.05 0.01 0.00 0.00 5 0.07 0.020.02 0.00 0.00 6 0.15 0.09 0.08 0.00 0.05

The f-values may be determined by solving an optimization with thelinear constrains discussed above. For example, referring to Table 1,the sum of column

${i = 1},{{{or}\mspace{14mu} {\sum\limits_{j = 1}^{n}{f_{ij}\mspace{14mu} {for}\mspace{14mu} 1}}} \leq i \leq m},{{equals}\mspace{14mu} p_{1}\mspace{11mu} {or}}\;,{{in}\mspace{14mu} {this}\mspace{14mu} {example}},{0.27.}$

Likewise, the sum of row

${j = 1},{{{or}\mspace{14mu} {\sum\limits_{i = 1}^{n}{f_{ij}\mspace{14mu} {for}\mspace{14mu} 1}}} \leq j \leq n},{{equals}\mspace{14mu} q_{1}\mspace{11mu} {or}}\;,{{in}\mspace{14mu} {this}\mspace{14mu} {example}},{0.15.}$

The Mallows distance may be determined by summing the valuesf_(ij)|x₁−y₁|^(α) for all values of i and j. FIG. 3 illustrates thesecalculations and the resulting sum, or Mallows distance, between thedistributions for product A and product B when α=2.

In an embodiment, the products may be grouped into one or more clustersbased on the corresponding similarity value. Products may be clusteredusing one or more clustering algorithms such as hierarchicalagglomerative clustering, K-means clustering or the like.

Hierarchical agglomerative clustering may be performed by regarding eachobject as a separate cluster, then merging these atomic clusters intolarger clusters until one or more predefined termination conditions aresatisfied. At each step, the two most similar objects (clusters orsingle object) may be identified and merged into a larger cluster.Deciding which two clusters are closest may be performed using a measureof the distance between each remaining pair of clusters. Such proximitymeasure is called a linkage metric. Major inter-cluster linkage metricsinclude single link, complete link and average link.

A single link metric may measure the similarity of two clusters based onthe distance between their closest (i.e., most similar) points. Thesingle link metric may often generate long straggle clusters.d(C₁,C₂)=min{d(x,y)|xεC₁,yεC₂}.

A complete link metric may measure the similarity of two clusters basedon the similarity of their most distant (i.e., least similar) points.The complete link metric may tend to form compact clusters.d(C₁,C₂)=max{d(x,y)|xεC₁,yεC₂}.

An average link metric may measure the similarity of two clusters basedon the average similarity of the points contained in the clusters.d(C₁,C₂)=average{d(x,y)|xεC₁,yεC₂}.

The particular link metric used to measure similarity may have an effecton the clustering of the objects because different link metrics reflectdifferent measures of closeness and connectivity. In an embodiment,values for a plurality of link metrics may be determined. Vendor datamay be considered close to other vendor data, for example, if thedistance between the data for each vendor is less than the distancebetween the data for the vendor and data for any other vendor. Relative“closeness” may depend on the nature of the data. Other methods ofdetermining closeness may also be performed within the scope of thepresent disclosure.

FIG. 4 depicts an exemplary diagram of clusters formed usinghierarchical agglomerative clustering. As illustrated in FIG. 4, 38vendors were clustered based on the distance between the ratingdistributions for each vendor and/or cluster of vendors. Clusters may bedetermined by selecting a distance threshold between clusters. Clustersthat exceed this threshold may be determined to be distinct. Forexample, a distance threshold of 0.3 may result in a determination ofthree clusters: {1, . . . , 3}, {4, . . . , 18} and {19, . . . , 38}.Likewise, a distance threshold of 0.2 may result in a determination offour clusters: {1, . . . , 3}, {4, . . . , 18}, {19, . . . , 23} and{24, . . . , 38}. Different distance thresholds may result in adifferent number of clusters.

In an embodiment, an optimal threshold may be determined by selectingthe threshold that optimizes a measure of cluster separation andcompactness. The optimal threshold may result in clusters that aretightly arranged about a center and distant from every other cluster.

In an embodiment, K-means clustering may be performed by firstdetermining a value K equal to the number of clusters to find. Next, aset of initial cluster centers, x₁, . . . , x_(K), may be chosen. Thesemay be chosen at random or by using a heuristic. For each point orvendor x in the dataset, the distances from that point to each of thecenters may be computed: d_(i)=d(x,x_(i)), i=1, . . . , K. Vendor x maybe assigned to the cluster with the closest center. After all points orvendors have been assigned, each center may be re-determined bycomputing the medoid for each cluster. A medoid is a representativeobject of a data set determined by finding the center of a cluster andselecting the object that is closest to the center. After selecting themedoid, the distances between the medoid and the other points may bere-determined. For example, if the members of cluster i are determinedto be {x_(i1), . . . ,x_(in)}, the new center or medoid is the point orvendor y in the set which minimizes

$\sum\limits_{j = 1}^{n}{{d( {y,x_{ij}} )}.}$

The new centers for each cluster are used to assign all the points orvendors to the cluster with the closest center. The process is repeateduntil the cluster centers do not change after each iteration.

In an embodiment, a dendrogram of the determined clusters, such as thatillustrated in FIG. 4, may be displayed to a user. A graph of clusterrating distributions for each cluster, such as that illustrated in FIG.5, may also he displayed to a user. A cluster rating distribution may bedetermined by averaging the rating distributions of the items in thecluster. In an embodiment, a graph similar to that depicted in FIG. 5may be displayed when a user clicks on a dendrogram similar to thatdepicted in FIG. 4. Alternatively, a graph may be displayed with adendrogram. As illustrated by FIG. 5, a user may be provided with arating associated with a product and also a range of ratings associatedwith the product. For example, cluster 1 500 may comprise the bestproducts and/or services, in this example, vendors, which have a highaverage rating and ratings which are consistently high. Cluster 2 505may comprise vendors with average ratings and a small variance in therange of ratings. Cluster 3 510 may comprise vendors with a largeraverage rating range, and may therefore he considered the worst cluster.As such, a user may discern from the graph that the vendors in cluster 1500 received consistently high ratings, that the vendors in cluster 2505 received consistently average ratings and that the vendors incluster 3 510 received inconsistently average ratings.

FIG. 6 depicts an exemplary flow chart according to the disclosedembodiments. A plurality of evaluator ratings for each print-relateditem in a set of print-related items may be received 600 and the ratingsmay be used to determine 605 a rating distribution for eachprint-related item. The similarity between the rating distributions fortwo print-related items may be determined 610 and the print-relateditems may be divided 615 into clusters based on the similarity values. Acomparison of the clusters may be displayed 620 to a user.

FIG. 7 depicts a block diagram of exemplary internal hardware of acomputing device that may be used to contain or implement the programinstructions according to an embodiment. A computing device processesdata to perform one or more functions. A computing device may be anyprocessor-based device such as, for example, a server, a personalcomputer, a personal digital assistant, a web-enabled phone, a smartterminal and a dumb terminal. A computing device may also include, butis not limited to, a computer, cell phone, personal digital assistant,gaming system, and/or other electronic device capable of communicatingin a networked environment.

Referring to FIG. 7, a bus 700 serves as the main information highwayinterconnecting the other illustrated components of the hardware. CPU705 is the central processing unit of the system, performingcalculations and logic operations required to execute a program. Readonly memory (ROM) 710 and random access memory (RAM) 715 constituteexemplary memory devices.

A disk controller 720 interfaces with one or more optional disk drivesto the system bus 700. These disk drives may include, for example,external or internal DVD drives 725, CD ROM drives 730 or hard drives735. As indicated previously, these various disk drives and diskcontrollers are optional devices.

Program instructions may be stored in the ROM 710 and/or the RAM 715.Optionally, program instructions may be stored on a computer readablemedium such as a compact disk or a digital disk or other recordingmedium, a communications signal or a carrier wave.

An optional display interface 740 may permit information from the bus700 to be displayed on the display 745 in audio, graphic or alphanumericformat. Communication with external devices may occur using variouscommunication ports 750. An exemplary communication port 750 may heattached to a communications network, such as the Internet or anintranet.

In addition to the standard computer-type components, the hardware mayalso include an interface 755 which allows for receipt of data frominput devices such as a keyboard 760 or other input device 765 such as amouse, remote control, pointer and/or joystick.

An embedded system, such as a sub-system within a xerographic apparatus,may optionally be used to perform one, some or all of the operationsdescribed herein. Likewise, a multiprocessor system may optionally beused to perform one, some or all of the operations described herein.

In an embodiment, distances between rating distributions may bedisplayed via a graphical interface, such as display interface 740.

It will be appreciated that various of the above-disclosed and otherfeatures and functions, or alternatives thereof, may be desirablycombined into many other different systems or applications. Also thatvarious presently unforeseen or unanticipated alternatives,modifications, variations or improvements therein may be subsequentlymade by those skilled in the art which are also intended to beencompassed by the following claims.

1. A method of evaluating customer satisfaction with a plurality ofprint-related items, the method comprising: receiving, a plurality ofevaluator ratings for each print-related item in a set of print-relateditems, wherein each rating comprises an ordinal scale value; for eachprint-related item: determining, by a computing device, based on thereceived evaluator ratings, a rating distribution for the print-relateditem, and determining, by the computing device, a similarity valuebetween the rating distribution for the print-related item and for eachother print-related item in the set of print-related items; dividing theset of print-related items into one or more clusters based on thecorresponding similarity values, wherein each cluster comprises one ormore print-related items; and displaying a comparison of the one or moreclusters to a user, wherein the comparison comprises one or more of: acomparison of the similarity values between one or more ratingdistributions associated with the one or more print-related items in oneor more clusters, a comparison of rating variability between one or morerating distributions associated with the one or more print related itemsin one or more clusters, and a comparison of one or more average ratingdistributions associated with one or more clusters.
 2. The method ofclaim 1, further comprising: allowing the user to select a clusterhaving an average rating that exceeds an average rating threshold valueand a standard deviation that is less than a threshold standarddeviation value.
 3. The method of claim 2, further comprising: allowingthe user to select a print-related item from the selected cluster basedon the rating distribution associated with the print-related item. 4.The method of claim 1, wherein each print-related item in the set ofprint-related items is provided by a unique vendor.
 5. The method ofclaim 1, wherein determining a rating distribution comprises, for eachprint-related item: determining a number of evaluators who selected eachrating; determining a total number of evaluators who evaluated theprint-related item; and determining a rating distribution for theprint-related item by, for each rating, dividing the number ofevaluators who selected the rating by the total number of evaluators. 6.The method of claim 1, wherein determining a similarity value comprises:determining a value equal to the difference between: (i) the entropy ofa sum of a product of a weight and a first probability distribution anda product of the weight and a second probability distribution and (ii) asum of a product of the weight and the entropy of the second probabilitydistribution and a product of the weight and the entropy of the secondprobability distribution; wherein the entropy of a probabilitydistribution p=(p₁, p₂, . . . , p_(n)) equals −Σ_(i−1) ^(n)p₁ log(p₁),and the weight is based on information pertaining to the evaluator whorated each print-related item.
 7. The method of claim 1, whereindetermining a similarity value comprises: determining an Euclidiandistance between the print-related item and each other print-relateditem in the set of print-related items.
 8. The method of claim 1,wherein determining a similarity value comprises: determining a Mallowsdistance between the print-related item and each other print-relateditem in the set of print-related items.
 9. The method of claim 1,wherein dividing the set of print-related items comprises: clusteringthe plurality of print-related items into one or more clusters based onthe computed similarity values.
 10. The method of claim 9, wherein:clustering comprises performing hierarchical agglomerative clustering;and dividing the set of print-related items comprises determining one ormore of a single link metric, a complete link metric and an average linkmetric.
 11. The method of claim 9, wherein the clustering comprisesperforming K-means clustering.
 12. The method of claim 1 whereindisplaying a comparison comprises: displaying a graph representingdifferences between the similarity measurements for each print-relateditem.
 13. The method of claim 17 wherein displaying a comparisoncomprises: displaying a dendrogram representing differences between thesimilarity measurements for each print-related item.
 14. The method ofclaim 1, wherein displaying a comparison comprises: for each cluster,displaying a graph representing a cluster rating distribution.
 15. Asystem of evaluating a plurality of print-related items, the systemcomprising: a processor; a communication port in communication with theprocessor; and a processor-readable storage medium in communication withthe processor, wherein the processor-readable storage medium containsone or more programming instructions for performing a method ofevaluating customer satisfaction with a plurality of print-relateditems, the method comprising: receiving a plurality of evaluator ratingsfor each print-related item in a set of print-related items wherein eachrating comprises an ordinal scale value, for each print-related item:determining, based on the received evaluator ratings, a ratingdistribution for the print-related item, and determining a similarityvalue between the rating distribution for the print-related item and foreach other print-related item in the set of print-related items,dividing the set of print-related items into one or more clusters basedon the corresponding similarity values, wherein each cluster comprisesone or more print-related items, and displaying a comparison of the oneor more clusters to a user, wherein the comparison comprises one or moreof: a comparison of the similarity values between one or more ratingdistributions associated with the one or more print-related items in oneor more clusters, a comparison of rating variability between one or morerating distributions associated with the one or more print related itemsin one or more clusters, and a comparison of one or more average ratingdistributions associated with one or more clusters.
 16. The system ofclaim 15, further comprising one or more programming instructions for:allowing the user to select a cluster having an average rating thatexceeds an average rating threshold value and a standard deviation thatis less than a threshold standard deviation value.
 17. The system ofclaim 16, further comprising one or more programming instructions for:allowing the user to select a print-related item from the selectedcluster based on the rating distribution associated with theprint-related item.
 18. The system of claim 15, wherein the one or moreprogramming instructions for determining a rating distributioncomprises, for each print-related item, one or more programminginstructions for: determining a number of evaluators who selected eachrating; determining a total number of evaluators who evaluated theprint-related item; and determining a rating distribution for theprint-related item by, for each rating, dividing the number ofevaluators who selected the rating by the total number of evaluators.19. The system of claim 15, wherein the one or more programminginstructions for determining a similarity value comprises one or moreprogramming instructions for: determining a value equal to thedifference between: (i) the entropy of a sum of a product of a weightand a first probability distribution and a product of the weight and asecond probability distribution and (ii) a sum of a product of theweight and the entropy of the second probability distribution and aproduct of the weight and the entropy of the second probabilitydistribution; wherein the entropy of a probability distribution p=(p₁,p₂, . . . , p_(n)) equals −Σ_(i=1) ^(n)p₁ log(p₁), and wherein the weighis based on information pertaining to the evaluator who rated eachprint-related item.
 20. The system of claim 15, wherein the one or moreprogramming instructions for displaying a comparison comprises one ormore programming instructions for: displaying a graph representingdifferences between the similarity measurements for each print-relateditem.
 21. The system of claim 15, wherein the one or more programminginstructions for displaying a comparison comprises one or moreprogramming instructions for: displaying a dendrogram representingdifferences between the similarity measurements for each print-relateditem.
 22. The system of claim 15, wherein the one or more programminginstructions for displaying a comparison comprises one or moreprogramming instructions for: for each cluster, displaying a graphrepresenting a cluster rating distribution.